
pip install xgboost

Requirement already satisfied: xgboost in /Users/alperenunal/anaconda3/lib/python3.11/site-packages (2.0.2)
Requirement already satisfied: numpy in /Users/alperenunal/anaconda3/lib/python3.11/site-packages (from xgboost) (1.24.3)
Requirement already satisfied: scipy in /Users/alperenunal/anaconda3/lib/python3.11/site-packages (from xgboost) (1.11.1)
Note: you may need to restart the kernel to use updated packages.


import pandas as pd
import numpy as np
import plotly.express as px
import pandas as pd
import plotly.graph_objects as go
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.inspection import permutation_importance
from sklearn.feature_selection import mutual_info_regression
from sklearn.preprocessing import MinMaxScaler
from xgboost import XGBRegressor
from xgboost import plot_importance


btcusdt_spot = pd.read_csv("btcusdt_spot_velo_1hour_010121.csv")
ethusdt_spot = pd.read_csv("ethusdt_spot_velo_1hour_010121.csv")
btcusdt_futures = pd.read_csv("btcusdt_futures_velo_1hour_010121.csv")
ethusdt_futures = pd.read_csv("ethusdt_futures_velo_1hour_010121.csv")


# Loading manually created ETF data
etf_df = pd.read_csv('ETF_List_sorted.csv')
etf_df


df = etf_df
df.drop(columns=['Company', 'Trade'], inplace=True)

# The first row was removed because it was an ETF event that
# took place before the start date of the historical data we will analyze.
df = df.iloc[1:].reset_index(drop=True)

# Converting 'timestamp' column to datetime format.
df['timestamp'] = pd.to_datetime(df['timestamp'])

# Creating 'btc_etf' and 'eth_etf' columns based on 'Currency' column.
df['btc_etf'] = df['Currency'].apply(lambda x: 1 if x == 'BTC' else 0)
df['eth_etf'] = df['Currency'].apply(lambda x: 1 if x == 'ETH' else 0)

# Filtering data between the specified start and end dates.
start_date = '2021-01-01 00:00:00'
end_date = '2024-07-30 18:00:00'
df = df[(df['timestamp'] >= start_date) & (df['timestamp'] <= end_date)]

# The 'timestamp' column is unique by adding a small delta to duplicates.
df['timestamp'] += pd.to_timedelta(df.groupby('timestamp').cumcount(), unit='s')

# Resampling to hourly frequency with all missing hours filled with 0.
all_hours = pd.date_range(start=start_date, end=end_date, freq='H')
df = df.set_index('timestamp').reindex(all_hours, fill_value=0).reset_index()
df.rename(columns={'index': 'timestamp'}, inplace=True)

# For each day where btc_etf or eth_etf was 1, it remains 1 for all hours.
df['btc_etf'] = df.groupby(df['timestamp'].dt.date)['btc_etf'].transform('max')
df['eth_etf'] = df.groupby(df['timestamp'].dt.date)['eth_etf'].transform('max')

# 'timestamp' column to 'YYYY-MM-DD hh:mm:ss'
df['timestamp'] = df['timestamp'].dt.strftime('%Y-%m-%d %H:%M:%S')

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 31363 entries, 0 to 31362
Data columns (total 4 columns):
 #   Column     Non-Null Count  Dtype 
---  ------     --------------  ----- 
 0   timestamp  31363 non-null  object
 1   Currency   31363 non-null  object
 2   btc_etf    31363 non-null  int64 
 3   eth_etf    31363 non-null  int64 
dtypes: int64(2), object(2)
memory usage: 980.2+ KB


df['timestamp'] = pd.to_datetime(df['timestamp'])
df


df.drop(columns='Currency', inplace=True)
df.rename(columns={'timestamp': 'time'}, inplace=True)
df['time'] = pd.to_datetime(df['time'])
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 31363 entries, 0 to 31362
Data columns (total 3 columns):
 #   Column   Non-Null Count  Dtype         
---  ------   --------------  -----         
 0   time     31363 non-null  datetime64[ns]
 1   btc_etf  31363 non-null  int64         
 2   eth_etf  31363 non-null  int64         
dtypes: datetime64[ns](1), int64(2)
memory usage: 735.2 KB


# Monitoring the event day to ensure all the btc_etf data is '1'.
nov_15_rows = df[df['time'].dt.date == pd.to_datetime('2021-11-15').date()]
nov_15_rows


# Net Volume Delta ('NVD')
btcusdt_spot['NVD'] = btcusdt_spot['buy_coin_volume'] - btcusdt_spot['sell_coin_volume']
btcusdt_spot['CVD'] = btcusdt_spot['NVD'].cumsum()

ethusdt_spot['NVD'] = ethusdt_spot['buy_coin_volume'] - ethusdt_spot['sell_coin_volume']
ethusdt_spot['CVD'] = ethusdt_spot['NVD'].cumsum()

btcusdt_futures['NVD'] = btcusdt_futures['buy_coin_volume'] - btcusdt_futures['sell_coin_volume']
btcusdt_futures['CVD'] = btcusdt_futures['NVD'].cumsum()

ethusdt_futures['NVD'] = ethusdt_futures['buy_coin_volume'] - ethusdt_futures['sell_coin_volume']
ethusdt_futures['CVD'] = ethusdt_futures['NVD'].cumsum()


# Renaming Columns for Clarification
btcusdt_spot.columns = ['spot_btc_' + col if col != 'time' else col for col in btcusdt_spot.columns]
ethusdt_spot.columns = ['spot_eth_' + col if col != 'time' else col for col in ethusdt_spot.columns]
btcusdt_futures.columns = ['futures_btc_' + col if col != 'time' else col for col in btcusdt_futures.columns]
ethusdt_futures.columns = ['futures_eth_' + col if col != 'time' else col for col in ethusdt_futures.columns]


spot_df = pd.merge(btcusdt_spot, ethusdt_spot, on='time')
spot_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 31363 entries, 0 to 31362
Data columns (total 37 columns):
 #   Column                       Non-Null Count  Dtype  
---  ------                       --------------  -----  
 0   spot_btc_exchange            31363 non-null  object 
 1   spot_btc_coin                31363 non-null  object 
 2   spot_btc_product             31363 non-null  object 
 3   time                         31363 non-null  object 
 4   spot_btc_open_price          31363 non-null  float64
 5   spot_btc_high_price          31363 non-null  float64
 6   spot_btc_low_price           31363 non-null  float64
 7   spot_btc_close_price         31363 non-null  float64
 8   spot_btc_coin_volume         31363 non-null  float64
 9   spot_btc_dollar_volume       31363 non-null  float64
 10  spot_btc_buy_trades          31363 non-null  int64  
 11  spot_btc_sell_trades         31363 non-null  int64  
 12  spot_btc_total_trades        31363 non-null  int64  
 13  spot_btc_buy_coin_volume     31363 non-null  float64
 14  spot_btc_sell_coin_volume    31363 non-null  float64
 15  spot_btc_buy_dollar_volume   31363 non-null  float64
 16  spot_btc_sell_dollar_volume  31363 non-null  float64
 17  spot_btc_NVD                 31363 non-null  float64
 18  spot_btc_CVD                 31363 non-null  float64
 19  spot_eth_exchange            31363 non-null  object 
 20  spot_eth_coin                31363 non-null  object 
 21  spot_eth_product             31363 non-null  object 
 22  spot_eth_open_price          31363 non-null  float64
 23  spot_eth_high_price          31363 non-null  float64
 24  spot_eth_low_price           31363 non-null  float64
 25  spot_eth_close_price         31363 non-null  float64
 26  spot_eth_coin_volume         31363 non-null  float64
 27  spot_eth_dollar_volume       31363 non-null  float64
 28  spot_eth_buy_trades          31363 non-null  int64  
 29  spot_eth_sell_trades         31363 non-null  int64  
 30  spot_eth_total_trades        31363 non-null  int64  
 31  spot_eth_buy_coin_volume     31363 non-null  float64
 32  spot_eth_sell_coin_volume    31363 non-null  float64
 33  spot_eth_buy_dollar_volume   31363 non-null  float64
 34  spot_eth_sell_dollar_volume  31363 non-null  float64
 35  spot_eth_NVD                 31363 non-null  float64
 36  spot_eth_CVD                 31363 non-null  float64
dtypes: float64(24), int64(6), object(7)
memory usage: 8.9+ MB


# Unnecesarry Columns to Drop
columns_to_delete = ['spot_btc_exchange', 'spot_btc_coin', 'spot_btc_product',
'spot_eth_exchange', 'spot_eth_coin', 'spot_eth_product']
spot_df.drop(columns = columns_to_delete, inplace= True)


# Removing Bybit Data from Dataset Because only Binance Data wil be used
btcusdt_futures = btcusdt_futures[btcusdt_futures["futures_btc_exchange"] != 'bybit']
ethusdt_futures = ethusdt_futures[ethusdt_futures["futures_eth_exchange"] != 'bybit']


futures_df = pd.merge(btcusdt_futures, ethusdt_futures, on='time')


# Unnecesarry Columns to Drop
columns_to_delete_futures = ['futures_btc_exchange', 'futures_btc_coin', 'futures_btc_product',
                             'futures_eth_exchange', 'futures_eth_coin', 'futures_eth_product']

futures_df.drop(columns = columns_to_delete_futures, inplace= True)
futures_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 31363 entries, 0 to 31362
Data columns (total 63 columns):
 #   Column                                       Non-Null Count  Dtype  
---  ------                                       --------------  -----  
 0   time                                         31363 non-null  object 
 1   futures_btc_open_price                       31363 non-null  float64
 2   futures_btc_high_price                       31363 non-null  float64
 3   futures_btc_low_price                        31363 non-null  float64
 4   futures_btc_close_price                      31363 non-null  float64
 5   futures_btc_coin_volume                      31363 non-null  float64
 6   futures_btc_dollar_volume                    31363 non-null  float64
 7   futures_btc_buy_trades                       31363 non-null  int64  
 8   futures_btc_sell_trades                      31363 non-null  int64  
 9   futures_btc_total_trades                     31363 non-null  int64  
 10  futures_btc_buy_coin_volume                  31363 non-null  float64
 11  futures_btc_sell_coin_volume                 31363 non-null  float64
 12  futures_btc_buy_dollar_volume                31363 non-null  float64
 13  futures_btc_sell_dollar_volume               31363 non-null  float64
 14  futures_btc_coin_open_interest_high          31363 non-null  float64
 15  futures_btc_coin_open_interest_low           31363 non-null  float64
 16  futures_btc_coin_open_interest_close         31363 non-null  float64
 17  futures_btc_dollar_open_interest_high        31363 non-null  float64
 18  futures_btc_dollar_open_interest_low         31363 non-null  float64
 19  futures_btc_dollar_open_interest_close       31363 non-null  float64
 20  futures_btc_funding_rate                     31363 non-null  float64
 21  futures_btc_premium                          31360 non-null  float64
 22  futures_btc_buy_liquidations                 31363 non-null  int64  
 23  futures_btc_sell_liquidations                31363 non-null  int64  
 24  futures_btc_buy_liquidations_coin_volume     31363 non-null  float64
 25  futures_btc_sell_liquidations_coin_volume    31363 non-null  float64
 26  futures_btc_liquidations_coin_volume         31363 non-null  float64
 27  futures_btc_buy_liquidations_dollar_volume   31363 non-null  float64
 28  futures_btc_sell_liquidations_dollar_volume  31363 non-null  float64
 29  futures_btc_liquidations_dollar_volume       31363 non-null  float64
 30  futures_btc_NVD                              31363 non-null  float64
 31  futures_btc_CVD                              31363 non-null  float64
 32  futures_eth_open_price                       31363 non-null  float64
 33  futures_eth_high_price                       31363 non-null  float64
 34  futures_eth_low_price                        31363 non-null  float64
 35  futures_eth_close_price                      31363 non-null  float64
 36  futures_eth_coin_volume                      31363 non-null  float64
 37  futures_eth_dollar_volume                    31363 non-null  float64
 38  futures_eth_buy_trades                       31363 non-null  int64  
 39  futures_eth_sell_trades                      31363 non-null  int64  
 40  futures_eth_total_trades                     31363 non-null  int64  
 41  futures_eth_buy_coin_volume                  31363 non-null  float64
 42  futures_eth_sell_coin_volume                 31363 non-null  float64
 43  futures_eth_buy_dollar_volume                31363 non-null  float64
 44  futures_eth_sell_dollar_volume               31363 non-null  float64
 45  futures_eth_coin_open_interest_high          31363 non-null  float64
 46  futures_eth_coin_open_interest_low           31363 non-null  float64
 47  futures_eth_coin_open_interest_close         31363 non-null  float64
 48  futures_eth_dollar_open_interest_high        31363 non-null  float64
 49  futures_eth_dollar_open_interest_low         31363 non-null  float64
 50  futures_eth_dollar_open_interest_close       31363 non-null  float64
 51  futures_eth_funding_rate                     31363 non-null  float64
 52  futures_eth_premium                          31360 non-null  float64
 53  futures_eth_buy_liquidations                 31363 non-null  int64  
 54  futures_eth_sell_liquidations                31363 non-null  int64  
 55  futures_eth_buy_liquidations_coin_volume     31363 non-null  float64
 56  futures_eth_sell_liquidations_coin_volume    31363 non-null  float64
 57  futures_eth_liquidations_coin_volume         31363 non-null  float64
 58  futures_eth_buy_liquidations_dollar_volume   31363 non-null  float64
 59  futures_eth_sell_liquidations_dollar_volume  31363 non-null  float64
 60  futures_eth_liquidations_dollar_volume       31363 non-null  float64
 61  futures_eth_NVD                              31363 non-null  float64
 62  futures_eth_CVD                              31363 non-null  float64
dtypes: float64(52), int64(10), object(1)
memory usage: 15.1+ MB


# Merging Spot & Futures Data
merged_df = pd.merge(spot_df, futures_df, on='time')


merged_df


# Calculating futures to spot price ratio
merged_df['btc_futures_to_spot'] = merged_df['futures_btc_close_price'] / merged_df['spot_btc_close_price']
merged_df['eth_futures_to_spot'] = merged_df['futures_eth_close_price'] / merged_df['spot_eth_close_price']

# Changing 'time' column's data type to datetime
merged_df['time'] = pd.to_datetime(merged_df['time'])


merged_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 31363 entries, 0 to 31362
Data columns (total 95 columns):
 #   Column                                       Non-Null Count  Dtype         
---  ------                                       --------------  -----         
 0   time                                         31363 non-null  datetime64[ns]
 1   spot_btc_open_price                          31363 non-null  float64       
 2   spot_btc_high_price                          31363 non-null  float64       
 3   spot_btc_low_price                           31363 non-null  float64       
 4   spot_btc_close_price                         31363 non-null  float64       
 5   spot_btc_coin_volume                         31363 non-null  float64       
 6   spot_btc_dollar_volume                       31363 non-null  float64       
 7   spot_btc_buy_trades                          31363 non-null  int64         
 8   spot_btc_sell_trades                         31363 non-null  int64         
 9   spot_btc_total_trades                        31363 non-null  int64         
 10  spot_btc_buy_coin_volume                     31363 non-null  float64       
 11  spot_btc_sell_coin_volume                    31363 non-null  float64       
 12  spot_btc_buy_dollar_volume                   31363 non-null  float64       
 13  spot_btc_sell_dollar_volume                  31363 non-null  float64       
 14  spot_btc_NVD                                 31363 non-null  float64       
 15  spot_btc_CVD                                 31363 non-null  float64       
 16  spot_eth_open_price                          31363 non-null  float64       
 17  spot_eth_high_price                          31363 non-null  float64       
 18  spot_eth_low_price                           31363 non-null  float64       
 19  spot_eth_close_price                         31363 non-null  float64       
 20  spot_eth_coin_volume                         31363 non-null  float64       
 21  spot_eth_dollar_volume                       31363 non-null  float64       
 22  spot_eth_buy_trades                          31363 non-null  int64         
 23  spot_eth_sell_trades                         31363 non-null  int64         
 24  spot_eth_total_trades                        31363 non-null  int64         
 25  spot_eth_buy_coin_volume                     31363 non-null  float64       
 26  spot_eth_sell_coin_volume                    31363 non-null  float64       
 27  spot_eth_buy_dollar_volume                   31363 non-null  float64       
 28  spot_eth_sell_dollar_volume                  31363 non-null  float64       
 29  spot_eth_NVD                                 31363 non-null  float64       
 30  spot_eth_CVD                                 31363 non-null  float64       
 31  futures_btc_open_price                       31363 non-null  float64       
 32  futures_btc_high_price                       31363 non-null  float64       
 33  futures_btc_low_price                        31363 non-null  float64       
 34  futures_btc_close_price                      31363 non-null  float64       
 35  futures_btc_coin_volume                      31363 non-null  float64       
 36  futures_btc_dollar_volume                    31363 non-null  float64       
 37  futures_btc_buy_trades                       31363 non-null  int64         
 38  futures_btc_sell_trades                      31363 non-null  int64         
 39  futures_btc_total_trades                     31363 non-null  int64         
 40  futures_btc_buy_coin_volume                  31363 non-null  float64       
 41  futures_btc_sell_coin_volume                 31363 non-null  float64       
 42  futures_btc_buy_dollar_volume                31363 non-null  float64       
 43  futures_btc_sell_dollar_volume               31363 non-null  float64       
 44  futures_btc_coin_open_interest_high          31363 non-null  float64       
 45  futures_btc_coin_open_interest_low           31363 non-null  float64       
 46  futures_btc_coin_open_interest_close         31363 non-null  float64       
 47  futures_btc_dollar_open_interest_high        31363 non-null  float64       
 48  futures_btc_dollar_open_interest_low         31363 non-null  float64       
 49  futures_btc_dollar_open_interest_close       31363 non-null  float64       
 50  futures_btc_funding_rate                     31363 non-null  float64       
 51  futures_btc_premium                          31360 non-null  float64       
 52  futures_btc_buy_liquidations                 31363 non-null  int64         
 53  futures_btc_sell_liquidations                31363 non-null  int64         
 54  futures_btc_buy_liquidations_coin_volume     31363 non-null  float64       
 55  futures_btc_sell_liquidations_coin_volume    31363 non-null  float64       
 56  futures_btc_liquidations_coin_volume         31363 non-null  float64       
 57  futures_btc_buy_liquidations_dollar_volume   31363 non-null  float64       
 58  futures_btc_sell_liquidations_dollar_volume  31363 non-null  float64       
 59  futures_btc_liquidations_dollar_volume       31363 non-null  float64       
 60  futures_btc_NVD                              31363 non-null  float64       
 61  futures_btc_CVD                              31363 non-null  float64       
 62  futures_eth_open_price                       31363 non-null  float64       
 63  futures_eth_high_price                       31363 non-null  float64       
 64  futures_eth_low_price                        31363 non-null  float64       
 65  futures_eth_close_price                      31363 non-null  float64       
 66  futures_eth_coin_volume                      31363 non-null  float64       
 67  futures_eth_dollar_volume                    31363 non-null  float64       
 68  futures_eth_buy_trades                       31363 non-null  int64         
 69  futures_eth_sell_trades                      31363 non-null  int64         
 70  futures_eth_total_trades                     31363 non-null  int64         
 71  futures_eth_buy_coin_volume                  31363 non-null  float64       
 72  futures_eth_sell_coin_volume                 31363 non-null  float64       
 73  futures_eth_buy_dollar_volume                31363 non-null  float64       
 74  futures_eth_sell_dollar_volume               31363 non-null  float64       
 75  futures_eth_coin_open_interest_high          31363 non-null  float64       
 76  futures_eth_coin_open_interest_low           31363 non-null  float64       
 77  futures_eth_coin_open_interest_close         31363 non-null  float64       
 78  futures_eth_dollar_open_interest_high        31363 non-null  float64       
 79  futures_eth_dollar_open_interest_low         31363 non-null  float64       
 80  futures_eth_dollar_open_interest_close       31363 non-null  float64       
 81  futures_eth_funding_rate                     31363 non-null  float64       
 82  futures_eth_premium                          31360 non-null  float64       
 83  futures_eth_buy_liquidations                 31363 non-null  int64         
 84  futures_eth_sell_liquidations                31363 non-null  int64         
 85  futures_eth_buy_liquidations_coin_volume     31363 non-null  float64       
 86  futures_eth_sell_liquidations_coin_volume    31363 non-null  float64       
 87  futures_eth_liquidations_coin_volume         31363 non-null  float64       
 88  futures_eth_buy_liquidations_dollar_volume   31363 non-null  float64       
 89  futures_eth_sell_liquidations_dollar_volume  31363 non-null  float64       
 90  futures_eth_liquidations_dollar_volume       31363 non-null  float64       
 91  futures_eth_NVD                              31363 non-null  float64       
 92  futures_eth_CVD                              31363 non-null  float64       
 93  btc_futures_to_spot                          31363 non-null  float64       
 94  eth_futures_to_spot                          31363 non-null  float64       
dtypes: datetime64[ns](1), float64(78), int64(16)
memory usage: 22.7 MB


# Creating an interactive graph for detection the liquidation cascades
fig = px.line(merged_df, x='time', y='futures_btc_close_price',
title='Price Comparison', width=1000, height=600)
fig.show()


# Based on the 'futures_btc_close_price' column, Liquidation cascade intervals are:
liquidation_periods = [
('2021-01-09 13:00:00', '2021-01-11 15:00:00'),
('2021-01-19 16:00:00', '2021-01-22 00:00:00'),
('2021-02-21 18:00:00', '2021-02-28 17:00:00'),
('2021-03-21 12:00:00', '2021-03-05 08:00:00'),
('2021-03-13 20:00:00', '2021-03-16 05:00:00'),
('2021-03-18 15:00:00', '2021-03-25 15:00:00'),
('2021-04-14 06:00:00', '2021-04-25 21:00:00'),
('2021-05-09 03:00:00', '2021-05-23 16:00:00'),
('2021-06-15 17:00:00', '2021-06-22 13:00:00'),
('2021-09-07 02:00:00', '2021-09-21 22:00:00'),
('2021-10-20 15:00:00', '2021-10-28 00:00:00'),
('2021-11-10 17:00:00', '2021-11-19 03:00:00'),
('2021-12-01 15:00:00', '2021-12-04 11:00:00'),
('2021-12-27 17:00:00', '2022-01-24 12:00:00'),
('2022-02-10 17:00:00', '2022-02-24 05:00:00'),
('2022-03-02 14:00:00', '2022-03-07 19:00:00'),
('2022-03-09 15:00:00', '2022-03-13 22:00:00'),
('2022-03-28 18:00:00', '2022-04-12 19:00:00'),
('2022-05-04 19:00:00', '2022-05-12 05:00:00'),
('2022-06-06 21:00:00', '2022-06-18 20:00:00'),
('2022-08-15 05:00:00', '2022-08-19 23:00:00'),
('2022-09-13 10:00:00', '2022-09-19 08:00:00'),
('2022-11-05 03:00:00', '2022-11-09 21:00:00'),
('2023-02-21 06:00:00', '2023-03-10 10:00:00'),
('2023-08-14 16:00:00', '2023-08-19 06:00:00'),
('2024-01-11 14:00:00', '2024-01-23 14:00:00'),
('2024-03-14 06:00:00', '2024-03-20 05:00:00'),
('2024-03-31 23:00:00', '2024-04-02 15:00:00'),
('2024-04-08 11:00:00', '2024-04-17 16:00:00'),
('2024-04-24 04:00:00', '2024-05-01 15:00:00'),
('2024-06-07 11:00:00', '2024-06-24 19:00:00'),
('2024-07-01 17:00:00', '2024-07-05 04:00:00'),
]

# Creating 'liquidation_cascades' column and set initial value to 0
merged_df['liquidation_cascades'] = 0

# Checking liquidation cascade periods and set 'liquidation_cascades' value to 1 for the relevant dates
for start_date, end_date in liquidation_periods:
    mask = (merged_df['time'] >= start_date) & (merged_df['time'] <= end_date)
    merged_df.loc[mask, 'liquidation_cascades'] = 1

# Checking the results
print(merged_df[['time', 'futures_btc_close_price', 'liquidation_cascades']].tail(30))

                     time  futures_btc_close_price  liquidation_cascades
31333 2024-07-29 13:00:00                  69249.8                     0
31334 2024-07-29 14:00:00                  68200.1                     0
31335 2024-07-29 15:00:00                  68067.2                     0
31336 2024-07-29 16:00:00                  66921.0                     0
31337 2024-07-29 17:00:00                  66980.8                     0
31338 2024-07-29 18:00:00                  67389.7                     0
31339 2024-07-29 19:00:00                  67276.6                     0
31340 2024-07-29 20:00:00                  67348.4                     0
31341 2024-07-29 21:00:00                  67459.9                     0
31342 2024-07-29 22:00:00                  67195.3                     0
31343 2024-07-29 23:00:00                  66750.0                     0
31344 2024-07-30 00:00:00                  66572.0                     0
31345 2024-07-30 01:00:00                  66176.5                     0
31346 2024-07-30 02:00:00                  66396.6                     0
31347 2024-07-30 03:00:00                  66585.6                     0
31348 2024-07-30 04:00:00                  66778.2                     0
31349 2024-07-30 05:00:00                  66455.9                     0
31350 2024-07-30 06:00:00                  66733.2                     0
31351 2024-07-30 07:00:00                  66914.0                     0
31352 2024-07-30 08:00:00                  66740.0                     0
31353 2024-07-30 09:00:00                  66580.0                     0
31354 2024-07-30 10:00:00                  66545.7                     0
31355 2024-07-30 11:00:00                  66532.5                     0
31356 2024-07-30 12:00:00                  66635.4                     0
31357 2024-07-30 13:00:00                  66357.3                     0
31358 2024-07-30 14:00:00                  65779.4                     0
31359 2024-07-30 15:00:00                  66198.0                     0
31360 2024-07-30 16:00:00                  66150.2                     0
31361 2024-07-30 17:00:00                  65839.8                     0
31362 2024-07-30 18:00:00                  65619.9                     0


# Traces for the line graph
line_trace = go.Scatter(
x=merged_df['time'],
y=merged_df['futures_btc_close_price'],
mode='lines',
name='futures_btc_close_price',
line=dict(color='blue')
)
# Traces for the liquidation cascade points
liquidation_trace = go.Scatter(
x=merged_df[merged_df['liquidation_cascades'] == 1]['time'],
y=merged_df[merged_df['liquidation_cascades'] == 1]['futures_btc_close_price'],
mode='markers',
name='Liquidation Cascades',
marker=dict(color='red', size=6)
    )
# Creating the figure and add the traces
fig = go.Figure()
fig.add_trace(line_trace)
fig.add_trace(liquidation_trace)

fig.update_layout(
title='Price & Liquidation Cascades',
xaxis_title='time',
yaxis_title='futures_btc_close_price',
width=1200,
height=800
)
fig.show()


#Finalising the dataset with adding ETF dates
merged_df = pd.merge(merged_df, df, on='time')


Columns = ['time', 'spot_btc_coin_volume',  'spot_btc_dollar_volume', 'spot_btc_total_trades', 'spot_btc_CVD',
           'futures_btc_close_price', 'futures_btc_coin_volume', 'futures_btc_dollar_volume', 'futures_btc_total_trades',
           'futures_btc_coin_open_interest_close', 'futures_btc_funding_rate', 'futures_btc_liquidations_coin_volume',
           'futures_btc_CVD', 'spot_eth_coin_volume',  'spot_eth_dollar_volume', 'spot_eth_total_trades', 'spot_eth_CVD',
           'futures_eth_close_price', 'futures_eth_coin_volume', 'futures_eth_dollar_volume', 'futures_eth_total_trades',
           'futures_eth_coin_open_interest_close', 'futures_eth_funding_rate', 'futures_eth_liquidations_coin_volume',
           'futures_eth_CVD', 'eth_etf', 'btc_etf', 'liquidation_cascades', 'btc_futures_to_spot', 'eth_futures_to_spot'] 

prediction_df = merged_df[Columns]

prediction_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 31363 entries, 0 to 31362
Data columns (total 30 columns):
 #   Column                                Non-Null Count  Dtype         
---  ------                                --------------  -----         
 0   time                                  31363 non-null  datetime64[ns]
 1   spot_btc_coin_volume                  31363 non-null  float64       
 2   spot_btc_dollar_volume                31363 non-null  float64       
 3   spot_btc_total_trades                 31363 non-null  int64         
 4   spot_btc_CVD                          31363 non-null  float64       
 5   futures_btc_close_price               31363 non-null  float64       
 6   futures_btc_coin_volume               31363 non-null  float64       
 7   futures_btc_dollar_volume             31363 non-null  float64       
 8   futures_btc_total_trades              31363 non-null  int64         
 9   futures_btc_coin_open_interest_close  31363 non-null  float64       
 10  futures_btc_funding_rate              31363 non-null  float64       
 11  futures_btc_liquidations_coin_volume  31363 non-null  float64       
 12  futures_btc_CVD                       31363 non-null  float64       
 13  spot_eth_coin_volume                  31363 non-null  float64       
 14  spot_eth_dollar_volume                31363 non-null  float64       
 15  spot_eth_total_trades                 31363 non-null  int64         
 16  spot_eth_CVD                          31363 non-null  float64       
 17  futures_eth_close_price               31363 non-null  float64       
 18  futures_eth_coin_volume               31363 non-null  float64       
 19  futures_eth_dollar_volume             31363 non-null  float64       
 20  futures_eth_total_trades              31363 non-null  int64         
 21  futures_eth_coin_open_interest_close  31363 non-null  float64       
 22  futures_eth_funding_rate              31363 non-null  float64       
 23  futures_eth_liquidations_coin_volume  31363 non-null  float64       
 24  futures_eth_CVD                       31363 non-null  float64       
 25  eth_etf                               31363 non-null  int64         
 26  btc_etf                               31363 non-null  int64         
 27  liquidation_cascades                  31363 non-null  int64         
 28  btc_futures_to_spot                   31363 non-null  float64       
 29  eth_futures_to_spot                   31363 non-null  float64       
dtypes: datetime64[ns](1), float64(22), int64(7)
memory usage: 7.2 MB


# Setting the 'time' column as the index of the prediction_df DataFrame,
# enabling time-based indexing and facilitating time series data manipulation.
prediction_df = prediction_df.set_index('time')


# Creation Log-returns and Volatility feature
# Calculating hourly returns
prediction_df['log_returns'] = np.log(prediction_df['futures_btc_close_price'] / prediction_df['futures_btc_close_price'].shift(1))

# Calculating volatility using rolling standard deviation (24-hour window)
prediction_df['volatility'] = prediction_df['log_returns'].rolling(window=24).std()


# For log_returns target
# Correlation matrix and target correlation calculation
cor_btc = prediction_df
correlation_matrix = cor_btc.corr()
target_correlation = correlation_matrix['log_returns'].sort_values(ascending=False)

# Converting to DataFrame for better display
target_correlation_df = target_correlation.reset_index()
target_correlation_df.columns = ['Feature', 'Correlation with Log Returns']

# Displaying
target_correlation_df


prediction_df.info()

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 31363 entries, 2021-01-01 00:00:00 to 2024-07-30 18:00:00
Data columns (total 31 columns):
 #   Column                                Non-Null Count  Dtype  
---  ------                                --------------  -----  
 0   spot_btc_coin_volume                  31363 non-null  float64
 1   spot_btc_dollar_volume                31363 non-null  float64
 2   spot_btc_total_trades                 31363 non-null  int64  
 3   spot_btc_CVD                          31363 non-null  float64
 4   futures_btc_close_price               31363 non-null  float64
 5   futures_btc_coin_volume               31363 non-null  float64
 6   futures_btc_dollar_volume             31363 non-null  float64
 7   futures_btc_total_trades              31363 non-null  int64  
 8   futures_btc_coin_open_interest_close  31363 non-null  float64
 9   futures_btc_funding_rate              31363 non-null  float64
 10  futures_btc_liquidations_coin_volume  31363 non-null  float64
 11  futures_btc_CVD                       31363 non-null  float64
 12  spot_eth_coin_volume                  31363 non-null  float64
 13  spot_eth_dollar_volume                31363 non-null  float64
 14  spot_eth_total_trades                 31363 non-null  int64  
 15  spot_eth_CVD                          31363 non-null  float64
 16  futures_eth_close_price               31363 non-null  float64
 17  futures_eth_coin_volume               31363 non-null  float64
 18  futures_eth_dollar_volume             31363 non-null  float64
 19  futures_eth_total_trades              31363 non-null  int64  
 20  futures_eth_coin_open_interest_close  31363 non-null  float64
 21  futures_eth_funding_rate              31363 non-null  float64
 22  futures_eth_liquidations_coin_volume  31363 non-null  float64
 23  futures_eth_CVD                       31363 non-null  float64
 24  eth_etf                               31363 non-null  int64  
 25  btc_etf                               31363 non-null  int64  
 26  liquidation_cascades                  31363 non-null  int64  
 27  btc_futures_to_spot                   31363 non-null  float64
 28  eth_futures_to_spot                   31363 non-null  float64
 29  log_returns                           31362 non-null  float64
 30  volatility                            31339 non-null  float64
dtypes: float64(24), int64(7)
memory usage: 7.7 MB


prediction_df = prediction_df.dropna()


# Saving prediction_df
prediction_df.to_csv('prediction_df.csv', index=False)


# For log_returns target

X = prediction_df.drop(columns=['log_returns'])  
y = prediction_df['log_returns'] 

model = XGBRegressor(n_estimators=100, random_state=42)
model.fit(X, y)

plt.figure(figsize=(18, 16))
plot_importance(model)
plt.title('Feature Importance for Log Returns')
plt.show()

<Figure size 1800x1600 with 0 Axes>


perm_importance = permutation_importance(model, X, y, n_repeats=10, random_state=42)
perm_importance_df = pd.DataFrame({'Feature': X.columns, 'Importance': perm_importance.importances_mean})
perm_importance_df = perm_importance_df.sort_values(by='Importance', ascending=False)

perm_importance_df


mi = mutual_info_regression(X, y)

mi_importance = pd.DataFrame({'Feature': X.columns, 'Importance Log Returns': mi})
mi_importance = mi_importance.sort_values(by='Importance Log Returns', ascending=False)


print(mi_importance)

                                 Feature  Importance Log Returns
6              futures_btc_dollar_volume                0.285483
10  futures_btc_liquidations_coin_volume                0.285337
7               futures_btc_total_trades                0.255280
13                spot_eth_dollar_volume                0.226058
14                 spot_eth_total_trades                0.216307
12                  spot_eth_coin_volume                0.215852
5                futures_btc_coin_volume                0.206107
22  futures_eth_liquidations_coin_volume                0.190577
19              futures_eth_total_trades                0.171471
18             futures_eth_dollar_volume                0.167824
0                   spot_btc_coin_volume                0.166778
1                 spot_btc_dollar_volume                0.166540
2                  spot_btc_total_trades                0.149137
29                            volatility                0.139134
17               futures_eth_coin_volume                0.135602
3                           spot_btc_CVD                0.085274
11                       futures_btc_CVD                0.066029
23                       futures_eth_CVD                0.062902
8   futures_btc_coin_open_interest_close                0.062701
15                          spot_eth_CVD                0.060193
4                futures_btc_close_price                0.046716
20  futures_eth_coin_open_interest_close                0.043631
27                   btc_futures_to_spot                0.037968
16               futures_eth_close_price                0.026432
9               futures_btc_funding_rate                0.023495
28                   eth_futures_to_spot                0.021930
21              futures_eth_funding_rate                0.016357
26                  liquidation_cascades                0.008262
24                               eth_etf                0.000138
25                               btc_etf                0.000000


cor_btc = prediction_df
correlation_matrix = cor_btc.corr()
target_correlation = correlation_matrix['futures_btc_close_price'].sort_values(ascending=False)

target_correlation_df = target_correlation.reset_index()
target_correlation_df.columns = ['Feature', 'Correlation with BTC Futures Price']
b
target_correlation_df


from xgboost import XGBRegressor
from xgboost import plot_importance
import matplotlib.pyplot as plt

# Define the input features and target variable
X = prediction_df.drop(columns=['futures_btc_close_price'])  
y = prediction_df['futures_btc_close_price']  # Target variable


model = XGBRegressor(n_estimators=100, random_state=42)
model.fit(X, y)


plt.figure(figsize=(18, 16))
plot_importance(model)
plt.title('Feature Importance for BTC Futures Price')
plt.show()

<Figure size 1800x1600 with 0 Axes>


perm_importance = permutation_importance(model, X, y, n_repeats=10, random_state=42)
perm_importance_df = pd.DataFrame({'Feature': X.columns, 'Importance': perm_importance.importances_mean})
perm_importance_df = perm_importance_df.sort_values(by='Importance', ascending=False)
perm_importance_df


mi = mutual_info_regression(X, y)

mi_importance = pd.DataFrame({'Feature': X.columns, 'Importance BTC Futures Price': mi})
mi_importance = mi_importance.sort_values(by='Importance BTC Futures Price', ascending=False)

print(mi_importance)

                                 Feature  Importance BTC Futures Price
3                           spot_btc_CVD                      3.118767
22                       futures_eth_CVD                      2.793937
10                       futures_btc_CVD                      2.665262
14                          spot_eth_CVD                      2.497796
15               futures_eth_close_price                      2.212861
7   futures_btc_coin_open_interest_close                      1.552230
19  futures_eth_coin_open_interest_close                      1.516246
29                            volatility                      0.805033
2                  spot_btc_total_trades                      0.463585
20              futures_eth_funding_rate                      0.450078
26                   btc_futures_to_spot                      0.442779
8               futures_btc_funding_rate                      0.435737
27                   eth_futures_to_spot                      0.411590
0                   spot_btc_coin_volume                      0.409441
1                 spot_btc_dollar_volume                      0.295178
12                spot_eth_dollar_volume                      0.222542
13                 spot_eth_total_trades                      0.212824
11                  spot_eth_coin_volume                      0.178857
16               futures_eth_coin_volume                      0.177275
5              futures_btc_dollar_volume                      0.148076
18              futures_eth_total_trades                      0.140082
4                futures_btc_coin_volume                      0.139650
9   futures_btc_liquidations_coin_volume                      0.127571
17             futures_eth_dollar_volume                      0.126356
6               futures_btc_total_trades                      0.122911
21  futures_eth_liquidations_coin_volume                      0.096202
25                  liquidation_cascades                      0.082224
28                           log_returns                      0.046752
24                               btc_etf                      0.012384
23                               eth_etf                      0.009579


# Preserve the original time series
time_index = prediction_df.index

# Create the MinMaxScaler
scaler = MinMaxScaler()

# Apply MinMax scaling to prediction_df (excluding the index)
scaled_data = scaler.fit_transform(prediction_df)

# Store the scaled data as a DataFrame
minmax_df = pd.DataFrame(scaled_data, columns=prediction_df.columns, index=time_index)

# The time series has been re-added as the index
minmax_df.index.name = 'time'


import matplotlib.pyplot as plt

# A figure is created with a specified size
plt.figure(figsize=(14, 8))

# The scaled BTC futures close price
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')

# The scaled ETH futures close price
plt.plot(minmax_df.index, minmax_df['futures_eth_close_price'], color='green', linewidth=1, alpha=0.5, label='futures_eth_close_price')

# Points where liquidation cascades occur are highlighted by scattering red dots on the BTC futures close price
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # The 's' value is increased for larger dots

# A title is added to the plot
plt.title('BTC Price and ETH Price with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')

# A legend is added below the plot, centered.
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['spot_btc_CVD'], color='green', linewidth=1, alpha=0.5, label='spot_btc_CVD')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots


plt.title('BTC Price and spot_btc_CVD with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['futures_btc_CVD'], color='green', linewidth=1, alpha=0.5, label='futures_btc_CVD')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots


plt.title('BTC Price and futures_btc_CVD with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['futures_btc_coin_open_interest_close'], color='green', linewidth=1, alpha=0.5, label='futures_btc_coin_open_interest_close')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots

plt.title('BTC Price and futures_btc_coin_open_interest_close with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['spot_eth_CVD'], color='green', linewidth=1, alpha=0.5, label='spot_eth_CVD')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots

plt.title('BTC Price and spot_eth_CVD with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['futures_eth_coin_open_interest_close'], color='green', linewidth=1, alpha=0.5, label='futures_eth_coin_open_interest_close')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots

plt.title('BTC Price and futures_eth_coin_open_interest_close with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['spot_btc_dollar_volume'], color='green', linewidth=1, alpha=0.5, label='spot_btc_dollar_volume')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots


plt.title('BTC Price and spot_btc_dollar_volume with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['futures_btc_dollar_volume'], color='green', linewidth=1, alpha=0.5, label='futures_btc_dollar_volume')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots


plt.title('BTC Price and futures_btc_dollar_volume with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['spot_eth_dollar_volume'], color='green', linewidth=1, alpha=0.5, label='spot_eth_dollar_volume')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots


plt.title('BTC Price and spot_eth_dollar_volume with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['spot_eth_total_trades'], color='green', linewidth=1, alpha=0.5, label='spot_eth_total_trades')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots


plt.title('BTC Price and spot_eth_total_trades with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['futures_eth_total_trades'], color='green', linewidth=1, alpha=0.5, label='futures_eth_total_trades')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots


plt.title('BTC Price and futures_eth_total_trades with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['spot_eth_coin_volume'], color='green', linewidth=1, alpha=0.5, label='spot_eth_coin_volume')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots

plt.title('BTC Price and spot_eth_coin_volume with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['futures_btc_coin_volume'], color='green', linewidth=1, alpha=0.5, label='futures_btc_coin_volume')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots

plt.title('BTC Price and futures_btc_coin_volume with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['futures_eth_coin_volume'], color='green', linewidth=1, alpha=0.5, label='futures_eth_coin_volume')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots


plt.title('BTC Price and futures_eth_coin_volume with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['spot_btc_total_trades'], color='green', linewidth=1, alpha=0.5, label='spot_btc_total_trades')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots


plt.title('BTC Price and spot_btc_total_trades with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['spot_btc_coin_volume'], color='green', linewidth=1, alpha=0.5, label='spot_btc_coin_volume')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots

plt.title('BTC Price and spot_btc_coin_volume with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['futures_eth_CVD'], color='green', linewidth=1, alpha=0.5, label='futures_eth_CVD')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots

plt.title('BTC Price and futures_eth_CVD with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


import matplotlib.pyplot as plt


plt.figure(figsize=(14, 8))


plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')


plt.plot(minmax_df.index, minmax_df['futures_btc_CVD'], color='green', linewidth=1, alpha=0.5, label='futures_btc_CVD')

plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots


plt.title('BTC Price and futures_btc_CVD with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')


plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)


plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['futures_btc_coin_open_interest_close'], color='green', linewidth=1, alpha=0.5, label='futures_btc_coin_open_interest_close')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots


plt.title('BTC Price and futures_btc_coin_open_interest_close with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['futures_btc_liquidations_coin_volume'], color='green', linewidth=1, alpha=0.5, label='futures_btc_liquidations_coin_volume')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots


plt.title('BTC Price and futures_btc_liquidations_coin_volume with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['btc_etf'], color='green', linewidth=1, alpha=0.5, label='btc_etf')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots

plt.title('BTC Price and btc_etf with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['eth_etf'], color='green', linewidth=1, alpha=0.5, label='eth_etf')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots


plt.title('BTC Price and eth_etf with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots

plt.title('BTC Price and eth_etf with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['futures_btc_funding_rate'], color='green', linewidth=1, alpha=0.5, label='futures_btc_funding_rate')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots

plt.title('BTC Price and futures_btc_funding_rate with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['futures_eth_funding_rate'], color='green', linewidth=1, alpha=0.5, label='futures_eth_funding_rate')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots

plt.title('BTC Price and futures_eth_funding_rate with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['btc_futures_to_spot'], color='green', linewidth=1, alpha=0.5, label='btc_futures_to_spot')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots


plt.title('BTC Price and btc_futures_to_spot with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


plt.figure(figsize=(14, 8))
plt.plot(minmax_df.index, minmax_df['futures_btc_close_price'], color='blue', linewidth=1, label='futures_btc_close_price')
plt.plot(minmax_df.index, minmax_df['eth_futures_to_spot'], color='green', linewidth=1, alpha=0.5, label='eth_futures_to_spot')
plt.scatter(minmax_df[minmax_df['liquidation_cascades'] == 1].index, 
            minmax_df[minmax_df['liquidation_cascades'] == 1]['futures_btc_close_price'], 
            color='red', s=30, label='Liquidation Cascades')  # Increased 's' value for larger dots


plt.title('BTC Price and eth_futures_to_spot with Liquidation Cascades')
plt.xlabel('Time')
plt.ylabel('Scaled Price')
plt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.1), fancybox=True, shadow=True, ncol=3)
plt.show()


minmax_scaled_price = minmax_df.drop(columns='futures_eth_close_price', inplace=False)
minmax_scaled_price.to_csv('minmax_scaled_price.csv', index=False)


prediction_scaled_price = prediction_df.drop(columns='futures_eth_close_price', inplace=False)
prediction_scaled_price.to_csv('prediction_scaled_price.csv', index=False)


columns_to_delete = ['futures_btc_dollar_volume', 'futures_btc_liquidations_coin_volume',
                     'spot_eth_dollar_volume', 'spot_eth_total_trades', 'futures_eth_total_trades',
                     'spot_eth_coin_volume', 'futures_eth_coin_volume', 'spot_btc_total_trades',
                     'futures_btc_liquidations_coin_volume', 'futures_eth_liquidations_coin_volume',
                     'futures_eth_dollar_volume', 'futures_btc_total_trades', 'spot_btc_CVD', 'spot_eth_CVD',
                     'spot_btc_dollar_volume']

prediction_df= prediction_df.drop(columns = columns_to_delete, errors='ignore')
minmax_df= minmax_df.drop(columns = columns_to_delete, errors='ignore')


minmax_df.to_csv('minmax_scaled_price.csv', index=False)


prediction_df.to_csv('prediction_scaled_price.csv', index=False)


cor_btc = prediction_df
correlation_matrix = cor_btc.corr()
target_correlation = correlation_matrix['futures_btc_close_price'].sort_values(ascending=False)

with pd.option_context('display.max_rows', None):
    print(target_correlation)

futures_btc_close_price                 1.000000
futures_eth_close_price                 0.822552
btc_futures_to_spot                     0.386361
futures_btc_funding_rate                0.320078
futures_eth_funding_rate                0.286436
eth_futures_to_spot                     0.282220
liquidation_cascades                    0.221141
volatility                              0.087102
log_returns                             0.010335
eth_etf                                 0.006500
btc_etf                                 0.005272
futures_btc_coin_volume                -0.224072
futures_eth_coin_open_interest_close   -0.398240
spot_btc_coin_volume                   -0.414169
futures_eth_CVD                        -0.426634
futures_btc_CVD                        -0.434872
futures_btc_coin_open_interest_close   -0.621279
Name: futures_btc_close_price, dtype: float64


correlation_matrix = prediction_df.corr()


plt.figure(figsize=(12, 10))
sns.heatmap(correlation_matrix, annot=True, fmt=".2f", cmap='coolwarm', linewidths=0.5)
plt.title('Correlation Heatmap of prediction_df')
plt.show()


columns_to_delete = ['futures_btc_dollar_volume', 'futures_btc_liquidations_coin_volume',
                     'spot_eth_dollar_volume', 'spot_eth_total_trades', 'futures_eth_total_trades',
                     'spot_eth_coin_volume', 'futures_eth_coin_volume', 'spot_btc_total_trades',
                     'futures_btc_liquidations_coin_volume', 'futures_eth_liquidations_coin_volume',
                     'futures_eth_dollar_volume', 'futures_btc_total_trades', 'spot_btc_CVD', 'spot_eth_CVD',
                     'spot_btc_dollar_volume']

prediction_df= prediction_df.drop(columns = columns_to_delete, errors='ignore')
minmax_df= minmax_df.drop(columns = columns_to_delete, errors='ignore')


minmax_df.to_csv('minmax_scaled_log_returns.csv', index=False)


prediction_df.to_csv('prediction_scaled_log_returns.csv', index=False)


log_returns = pd.read_csv("prediction_scaled_log_returns.csv")
price = pd.read_csv("prediction_scaled_price.csv")


log_returns.columns

Index(['spot_btc_coin_volume', 'futures_btc_close_price',
       'futures_btc_coin_volume', 'futures_btc_coin_open_interest_close',
       'futures_btc_funding_rate', 'futures_btc_CVD',
       'futures_eth_close_price', 'futures_eth_coin_open_interest_close',
       'futures_eth_funding_rate', 'futures_eth_CVD', 'eth_etf', 'btc_etf',
       'liquidation_cascades', 'btc_futures_to_spot', 'eth_futures_to_spot',
       'log_returns', 'volatility'],
      dtype='object')


price.columns

Index(['spot_btc_coin_volume', 'futures_btc_close_price',
       'futures_btc_coin_volume', 'futures_btc_coin_open_interest_close',
       'futures_btc_funding_rate', 'futures_btc_CVD',
       'futures_eth_close_price', 'futures_eth_coin_open_interest_close',
       'futures_eth_funding_rate', 'futures_eth_CVD', 'eth_etf', 'btc_etf',
       'liquidation_cascades', 'btc_futures_to_spot', 'eth_futures_to_spot',
       'log_returns', 'volatility'],
      dtype='object')


prediction_df = pd.read_csv("prediction_scaled_log_returns.csv")


prediction_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 31339 entries, 0 to 31338
Data columns (total 17 columns):
 #   Column                                Non-Null Count  Dtype  
---  ------                                --------------  -----  
 0   spot_btc_coin_volume                  31339 non-null  float64
 1   futures_btc_close_price               31339 non-null  float64
 2   futures_btc_coin_volume               31339 non-null  float64
 3   futures_btc_coin_open_interest_close  31339 non-null  float64
 4   futures_btc_funding_rate              31339 non-null  float64
 5   futures_btc_CVD                       31339 non-null  float64
 6   futures_eth_close_price               31339 non-null  float64
 7   futures_eth_coin_open_interest_close  31339 non-null  float64
 8   futures_eth_funding_rate              31339 non-null  float64
 9   futures_eth_CVD                       31339 non-null  float64
 10  eth_etf                               31339 non-null  int64  
 11  btc_etf                               31339 non-null  int64  
 12  liquidation_cascades                  31339 non-null  int64  
 13  btc_futures_to_spot                   31339 non-null  float64
 14  eth_futures_to_spot                   31339 non-null  float64
 15  log_returns                           31339 non-null  float64
 16  volatility                            31339 non-null  float64
dtypes: float64(14), int64(3)
memory usage: 4.1 MB


start_date = '2021-01-02 00:00:00'
end_date = '2024-07-30 18:00:00'
freq = 'H'  
datetime_index = pd.date_range(start=start_date, end=end_date, freq=freq)
prediction_df = prediction_df.set_index(datetime_index)


prediction_df.info()

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 31339 entries, 2021-01-02 00:00:00 to 2024-07-30 18:00:00
Freq: H
Data columns (total 17 columns):
 #   Column                                Non-Null Count  Dtype  
---  ------                                --------------  -----  
 0   spot_btc_coin_volume                  31339 non-null  float64
 1   futures_btc_close_price               31339 non-null  float64
 2   futures_btc_coin_volume               31339 non-null  float64
 3   futures_btc_coin_open_interest_close  31339 non-null  float64
 4   futures_btc_funding_rate              31339 non-null  float64
 5   futures_btc_CVD                       31339 non-null  float64
 6   futures_eth_close_price               31339 non-null  float64
 7   futures_eth_coin_open_interest_close  31339 non-null  float64
 8   futures_eth_funding_rate              31339 non-null  float64
 9   futures_eth_CVD                       31339 non-null  float64
 10  eth_etf                               31339 non-null  int64  
 11  btc_etf                               31339 non-null  int64  
 12  liquidation_cascades                  31339 non-null  int64  
 13  btc_futures_to_spot                   31339 non-null  float64
 14  eth_futures_to_spot                   31339 non-null  float64
 15  log_returns                           31339 non-null  float64
 16  volatility                            31339 non-null  float64
dtypes: float64(14), int64(3)
memory usage: 4.3 MB


import pandas as pd
import matplotlib.pyplot as plt
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf

# Columns to be used (excluding binary columns)
columns_to_use = [
    'spot_btc_coin_volume', 'futures_btc_coin_volume',
    'futures_btc_coin_open_interest_close', 'futures_btc_funding_rate',
    'futures_btc_CVD', 'futures_eth_coin_open_interest_close',
    'futures_eth_funding_rate', 'futures_eth_CVD',
    'btc_futures_to_spot', 'eth_futures_to_spot', 'log_returns',
    'volatility'
]

# Filtering the data for ACF and PACF analysis
data_to_analyze = prediction_df[columns_to_use]

# Plotting ACF and PACF graphs for each column
for column in data_to_analyze.columns:
    print(f'ACF and PACF Plots - {column}')
    
    # ACF Plot
    plt.figure(figsize=(10, 5))
    plot_acf(data_to_analyze[column], lags=1440)
    plt.title(f'ACF for {column}')
    plt.show()
    
    # PACF Plot
    plt.figure(figsize=(10, 5))
    plot_pacf(data_to_analyze[column], lags=1440)
    plt.title(f'PACF for {column}')
    plt.show()

ACF and PACF Plots - spot_btc_coin_volume

<Figure size 1000x500 with 0 Axes>

<Figure size 1000x500 with 0 Axes>

ACF and PACF Plots - futures_btc_coin_volume

<Figure size 1000x500 with 0 Axes>

<Figure size 1000x500 with 0 Axes>

ACF and PACF Plots - futures_btc_coin_open_interest_close

<Figure size 1000x500 with 0 Axes>

<Figure size 1000x500 with 0 Axes>

ACF and PACF Plots - futures_btc_funding_rate


predictions_daily = prediction_df.resample('D').last()


# log_returns and volatility features have to be recalculated. Dataset's frequency was changed as daily,
# for that reason, log_returns and volatility have to be calculated daily.
predictions_daily = predictions_daily.drop(columns=['log_returns', 'volatility'])


predictions_daily['log_returns'] = np.log(predictions_daily['futures_btc_close_price'] / predictions_daily['futures_btc_close_price'].shift(1))

predictions_daily['volatility'] = predictions_daily['log_returns'].rolling(window=10).std()


predictions_daily.info()

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 1306 entries, 2021-01-02 to 2024-07-30
Freq: D
Data columns (total 17 columns):
 #   Column                                Non-Null Count  Dtype  
---  ------                                --------------  -----  
 0   spot_btc_coin_volume                  1306 non-null   float64
 1   futures_btc_close_price               1306 non-null   float64
 2   futures_btc_coin_volume               1306 non-null   float64
 3   futures_btc_coin_open_interest_close  1306 non-null   float64
 4   futures_btc_funding_rate              1306 non-null   float64
 5   futures_btc_CVD                       1306 non-null   float64
 6   futures_eth_close_price               1306 non-null   float64
 7   futures_eth_coin_open_interest_close  1306 non-null   float64
 8   futures_eth_funding_rate              1306 non-null   float64
 9   futures_eth_CVD                       1306 non-null   float64
 10  eth_etf                               1306 non-null   int64  
 11  btc_etf                               1306 non-null   int64  
 12  liquidation_cascades                  1306 non-null   int64  
 13  btc_futures_to_spot                   1306 non-null   float64
 14  eth_futures_to_spot                   1306 non-null   float64
 15  log_returns                           1305 non-null   float64
 16  volatility                            1296 non-null   float64
dtypes: float64(14), int64(3)
memory usage: 183.7 KB


predictions_daily = predictions_daily.dropna()


# Features to be excluded: 'eth_etf', 'btc_etf', 'liquidation_cascades'
excluded_features = ['eth_etf', 'btc_etf', 'liquidation_cascades']
target = 'log_returns'

# Selecting the features to include in the test
features = [col for col in predictions_daily.columns if col not in excluded_features and col != target]

# Granger Causality Test
maxlag = 60  # 60-day lag (approximately 2 months)

# Dictionary to store the results
results = {}

# Performing Granger Causality Test for each feature
for feature in features:
    print(f"\nTesting Granger Causality between {feature} and {target}")
    test_result = grangercausalitytests(predictions_daily[[target, feature]], maxlag=maxlag, verbose=False)
    results[feature] = test_result

# Reviewing the results
for feature, result in results.items():
    print(f"\nFeature: {feature}")
    for lag in result:
        print(f"Lag {lag} p-value: {result[lag][0]['ssr_ftest'][1]:.4f}")  # F-test p-value

Testing Granger Causality between spot_btc_coin_volume and log_returns

/Users/alperenunal/anaconda3/lib/python3.11/site-packages/statsmodels/tsa/stattools.py:1488: FutureWarning: verbose is deprecated since functions should not print results
  warnings.warn(

Testing Granger Causality between futures_btc_close_price and log_returns

/Users/alperenunal/anaconda3/lib/python3.11/site-packages/statsmodels/tsa/stattools.py:1488: FutureWarning: verbose is deprecated since functions should not print results
  warnings.warn(

Testing Granger Causality between futures_btc_coin_volume and log_returns

/Users/alperenunal/anaconda3/lib/python3.11/site-packages/statsmodels/tsa/stattools.py:1488: FutureWarning: verbose is deprecated since functions should not print results
  warnings.warn(

Testing Granger Causality between futures_btc_coin_open_interest_close and log_returns

/Users/alperenunal/anaconda3/lib/python3.11/site-packages/statsmodels/tsa/stattools.py:1488: FutureWarning: verbose is deprecated since functions should not print results
  warnings.warn(

Testing Granger Causality between futures_btc_funding_rate and log_returns

/Users/alperenunal/anaconda3/lib/python3.11/site-packages/statsmodels/tsa/stattools.py:1488: FutureWarning: verbose is deprecated since functions should not print results
  warnings.warn(


# Lagged cross-correlation function
def lagged_cross_correlation(x, y, max_lag):
    result = [x.corr(y.shift(lag)) for lag in range(-max_lag, max_lag + 1)]
    return np.array(result)

# Parameters
max_lag = 60  # Considering 60 lags as in the Granger causality test
target = 'log_returns'
features = ['spot_btc_coin_volume', 'futures_btc_coin_volume',
    'futures_btc_coin_open_interest_close', 'futures_btc_funding_rate',
    'futures_btc_CVD', 'futures_eth_coin_open_interest_close',
    'futures_eth_funding_rate', 'futures_eth_CVD',
    'btc_futures_to_spot', 'eth_futures_to_spot', 'futures_btc_close_price',
    'volatility']

# Performing Lagged Cross-Correlation Analysis
correlation_results = {}

for feature in features:
    cross_corr = lagged_cross_correlation(predictions_daily[feature], predictions_daily[target], max_lag)
    correlation_results[feature] = cross_corr

    # Plotting the results
    plt.figure(figsize=(8, 4))
    plt.plot(range(-max_lag, max_lag + 1), cross_corr)
    plt.title(f'Lagged Cross-Correlation between {feature} and {target}')
    plt.xlabel('Lag')
    plt.ylabel('Cross-Correlation')
    plt.axhline(0, color='black', linestyle='--', linewidth=1)
    plt.show()

# Reviewing the results
correlation_results_df = pd.DataFrame(correlation_results, index=range(-max_lag, max_lag + 1))
correlation_results_df

	time	btc_etf
7632	2021-11-15 00:00:00	1
7633	2021-11-15 01:00:00	1
7634	2021-11-15 02:00:00	1
7635	2021-11-15 03:00:00	1
7636	2021-11-15 04:00:00	1
7637	2021-11-15 05:00:00	1
7638	2021-11-15 06:00:00	1
7639	2021-11-15 07:00:00	1
7640	2021-11-15 08:00:00	1
7641	2021-11-15 09:00:00	1
7642	2021-11-15 10:00:00	1
7643	2021-11-15 11:00:00	1
7644	2021-11-15 12:00:00	1
7645	2021-11-15 13:00:00	1
7646	2021-11-15 14:00:00	1
7647	2021-11-15 15:00:00	1
7648	2021-11-15 16:00:00	1
7649	2021-11-15 17:00:00	1
7650	2021-11-15 18:00:00	1
7651	2021-11-15 19:00:00	1
7652	2021-11-15 20:00:00	1
7653	2021-11-15 21:00:00	1
7654	2021-11-15 22:00:00	1
7655	2021-11-15 23:00:00	1

	time	spot_btc_open_price	spot_btc_high_price	spot_btc_low_price	spot_btc_close_price	spot_btc_coin_volume	spot_btc_dollar_volume	spot_btc_buy_trades	spot_btc_sell_trades	spot_btc_total_trades	...	futures_eth_buy_liquidations	futures_eth_sell_liquidations	futures_eth_buy_liquidations_coin_volume	futures_eth_sell_liquidations_coin_volume	futures_eth_liquidations_coin_volume	futures_eth_buy_liquidations_dollar_volume	futures_eth_sell_liquidations_dollar_volume	futures_eth_liquidations_dollar_volume	futures_eth_NVD	futures_eth_CVD
0	2021-01-01 00:00:00	28975.65	29031.34	28690.17	28995.13	2128.921567	6.146804e+07	27613	25162	52775	...	5	33	21.972	279.905	301.877	1.615119e+04	204916.42667	2.210676e+05	-14060.299	-1.406030e+04
1	2021-01-01 01:00:00	28995.13	29470.00	28960.35	29409.99	5403.068471	1.583578e+08	59341	44555	103896	...	42	0	1861.541	0.000	1861.541	1.383821e+06	0.00000	1.383821e+06	1493.404	-1.381830e+04
2	2021-01-01 02:00:00	29409.99	29465.26	29120.03	29194.65	2384.231560	6.984265e+07	29051	28595	57646	...	2	7	0.376	29.946	30.322	2.808902e+02	22303.70485	2.258460e+04	-18465.699	-3.247673e+04
3	2021-01-01 03:00:00	29194.65	29367.00	29150.02	29278.40	1461.345077	4.276078e+07	22782	19728	42510	...	0	1	0.000	1.863	1.863	0.000000e+00	1387.58103	1.387581e+03	4070.918	-2.839819e+04
4	2021-01-01 04:00:00	29278.40	29395.00	29029.40	29220.31	2038.046803	5.961464e+07	27193	28221	55414	...	6	18	27.764	72.904	100.668	2.075376e+04	54193.04133	7.494680e+04	-10863.373	-3.965541e+04
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
31358	2024-07-30 14:00:00	66381.99	66449.99	65660.89	65810.00	2571.180050	1.699933e+08	60999	69820	130819	...	1	69	0.029	251.286	251.315	9.647100e+01	831160.05300	8.312565e+05	-4808.555	-3.346359e+07
31359	2024-07-30 15:00:00	65810.00	66332.01	65555.00	66216.01	1403.788890	9.259820e+07	54286	48539	102825	...	20	13	27.411	19.515	46.926	9.077219e+04	64262.31200	1.550345e+05	6774.840	-3.346003e+07
31360	2024-07-30 16:00:00	66216.01	66550.01	66145.00	66180.01	629.106070	4.172559e+07	30733	25702	56435	...	6	14	81.218	18.085	99.303	2.700290e+05	59828.56800	3.298576e+05	-6013.903	-3.345667e+07
31361	2024-07-30 17:00:00	66180.01	66210.01	65684.68	65868.01	1013.370530	6.679952e+07	42195	46843	89038	...	8	77	12.896	268.487	281.383	4.259585e+04	884233.73000	9.268296e+05	-11256.910	-3.346888e+07
31362	2024-07-30 18:00:00	65868.01	66074.98	65600.00	65730.00	764.145240	5.029318e+07	36161	34833	70994	...	3	30	2.361	67.421	69.782	7.780520e+03	221305.24500	2.290858e+05	-13051.165	-3.348799e+07

	spot_btc_coin_volume	futures_btc_coin_volume	futures_btc_coin_open_interest_close	futures_btc_funding_rate	futures_btc_CVD	futures_eth_coin_open_interest_close	futures_eth_funding_rate	futures_eth_CVD	btc_futures_to_spot	eth_futures_to_spot	futures_btc_close_price	volatility
-60	0.014233	0.033137	0.016550	-0.026041	-0.035220	0.032092	-0.015272	-0.012210	-0.077827	-0.003179	-0.050450	-0.000453
-59	0.013942	0.025818	0.016773	-0.032339	-0.035309	0.033034	-0.007351	-0.012947	0.021004	-0.009731	-0.049629	0.018221
-58	0.036573	0.057026	0.008890	-0.003345	-0.032296	0.031144	-0.010387	-0.009285	-0.031482	-0.007223	-0.046138	0.026090
-57	-0.025084	-0.046710	0.015462	-0.022825	-0.030019	0.031974	-0.023196	-0.007314	-0.009984	-0.008615	-0.042655	0.004531
-56	0.005471	-0.003002	0.018018	-0.021976	-0.026078	0.036448	0.062498	-0.002688	0.029085	0.036311	-0.040707	0.007882
...	...	...	...	...	...	...	...	...	...	...	...	...
56	0.017418	0.005297	-0.042958	0.070028	-0.030914	-0.027475	0.108292	-0.017932	0.044458	0.076263	0.063096	0.017158
57	0.013932	0.011532	-0.051296	0.082643	-0.028698	-0.029822	0.026879	-0.016653	0.049372	0.070354	0.065027	0.022192
58	0.006604	-0.024388	-0.046506	0.083757	-0.028028	-0.033877	0.012481	-0.016643	0.016932	0.036280	0.068065	0.032196
59	-0.000495	-0.031313	-0.055574	0.050194	-0.029419	-0.035908	0.045028	-0.018043	0.050128	0.059417	0.069290	0.034203
60	-0.002976	-0.007311	-0.058969	0.033134	-0.032349	-0.033739	0.015452	-0.019673	-0.002414	0.052512	0.063262	0.039023

	timestamp	Company	Currency	Trade
0	2017-11-22	Bitwise 10 Crypto Index Fund	ETH	Spot
1	2021-03-26	VanEck Ethereum Strategy ETF	ETH	Spot
2	2021-10-21	Valkyrie Bitcoin and Ether Strategy ETF	ETH	Futures
3	2021-11-15	Global X Blockchain & Bitcoin Strategy ETF	BTC	Futures
4	2021-11-18	ProShares Bitcoin Strategy ETF	BTC	Futures
5	2021-11-21	Valkyrie Bitcoin and Ether Strategy ETF	BTC	Futures
6	2022-09-15	Hashdex Bitcoin Futures ETF	BTC	Futures
7	2023-02-10	ProShares Ether Strategy ETF	ETH	Spot
8	2023-03-20	Bitwise Bitcoin Strategy Optimum Yield ETF	BTC	Futures
9	2023-09-29	Bitwise Ethereum Strategy ETF	ETH	Spot
10	2023-09-29	Bitwise Bitcoin and Ether Equal Weight Strateg...	ETH	Spot
11	2023-10-02	ProShares Bitcoin & Ether Equal Weight Strateg...	BTC	Futures
12	2023-10-02	ProShares Bitcoin & Ether Market Cap Weight St...	ETH	Futures
13	2023-10-02	ProShares Bitcoin & Ether Equal Weight Strateg...	ETH	Futures
14	2023-10-02	ProShares Bitcoin & Ether Market Cap Weight St...	BTC	Futures
15	2023-11-14	ARK 21Shares Active Ethereum Futures Strategy	ETH	Spot
16	2023-11-14	ARK 21Shares Active Bitcoin Futures Strategy ETF	BTC	Futures
17	2023-11-15	ARK 21Shares Active On-Chain Bitcoin Strategy ETF	BTC	Futures
18	2023-11-15	ARK 21Shares Active Bitcoin Ethereum Strategy ETF	BTC	Futures
19	2024-01-05	iShares Bitcoin Trust	BTC	Spot
20	2024-01-10	Bitwise Bitcoin ETP	BTC	Spot
21	2024-01-10	Ark/21 Shares Bitcoin Trust	BTC	Spot
22	2024-01-10	Valkyrie Bitcoin Fund	BTC	Spot
23	2024-01-11	Invesco Galaxy Bitcoin ETF	BTC	Spot
24	2024-01-11	VanEck Bitcoin Trust	BTC	Spot
25	2024-01-11	WisdomTree Bitcoin Trust	BTC	Spot
26	2024-01-11	Franklin Bitcoin ETF	BTC	Spot
27	2024-01-11	Wise Origin Bitcoin Trust by Fidelity	BTC	Spot
28	2024-01-11	Grayscale Bitcoin Trust	BTC	Spot

	Feature	Correlation with Log Returns
0	log_returns	1.000000
1	eth_futures_to_spot	0.026956
2	volatility	0.011714
3	futures_btc_close_price	0.010303
4	futures_eth_CVD	0.008693
5	futures_btc_CVD	0.007548
6	futures_eth_coin_open_interest_close	0.003631
7	futures_eth_close_price	0.002391
8	spot_eth_CVD	-0.000906
9	eth_etf	-0.002324
10	futures_btc_coin_open_interest_close	-0.002597
11	spot_btc_CVD	-0.002969
12	futures_eth_funding_rate	-0.004351
13	btc_futures_to_spot	-0.004866
14	btc_etf	-0.005193
15	spot_btc_coin_volume	-0.014327
16	spot_btc_total_trades	-0.016394
17	futures_btc_funding_rate	-0.019416
18	spot_btc_dollar_volume	-0.026889
19	futures_btc_coin_volume	-0.041359
20	futures_btc_total_trades	-0.051736
21	futures_btc_dollar_volume	-0.052388
22	futures_eth_coin_volume	-0.066181
23	spot_eth_total_trades	-0.067869
24	spot_eth_coin_volume	-0.070614
25	futures_btc_liquidations_coin_volume	-0.076852
26	futures_eth_total_trades	-0.078598
27	spot_eth_dollar_volume	-0.082632
28	futures_eth_dollar_volume	-0.086016
29	liquidation_cascades	-0.086660
30	futures_eth_liquidations_coin_volume	-0.109899

	Feature	Importance
27	btc_futures_to_spot	0.271289
4	futures_btc_close_price	0.233474
22	futures_eth_liquidations_coin_volume	0.211302
6	futures_btc_dollar_volume	0.191674
28	eth_futures_to_spot	0.180187
10	futures_btc_liquidations_coin_volume	0.137095
12	spot_eth_coin_volume	0.117724
18	futures_eth_dollar_volume	0.116423
16	futures_eth_close_price	0.111468
13	spot_eth_dollar_volume	0.108617
20	futures_eth_coin_open_interest_close	0.107966
9	futures_btc_funding_rate	0.107029
5	futures_btc_coin_volume	0.092551
26	liquidation_cascades	0.081697
7	futures_btc_total_trades	0.078206
29	volatility	0.077506
21	futures_eth_funding_rate	0.070549
19	futures_eth_total_trades	0.068990
15	spot_eth_CVD	0.068131
8	futures_btc_coin_open_interest_close	0.065895
0	spot_btc_coin_volume	0.064768
2	spot_btc_total_trades	0.062339
1	spot_btc_dollar_volume	0.059547
17	futures_eth_coin_volume	0.058410
3	spot_btc_CVD	0.056864
23	futures_eth_CVD	0.056547
14	spot_eth_total_trades	0.051055
11	futures_btc_CVD	0.050401
25	btc_etf	0.000063
24	eth_etf	0.000051

	Feature	Importance
15	futures_eth_close_price	0.719054
22	futures_eth_CVD	0.226792
3	spot_btc_CVD	0.114155
7	futures_btc_coin_open_interest_close	0.092609
10	futures_btc_CVD	0.025362
14	spot_eth_CVD	0.013361
26	btc_futures_to_spot	0.006731
19	futures_eth_coin_open_interest_close	0.003811
27	eth_futures_to_spot	0.001574
8	futures_btc_funding_rate	0.000750
20	futures_eth_funding_rate	0.000439
29	volatility	0.000378
2	spot_btc_total_trades	0.000259
5	futures_btc_dollar_volume	0.000175
25	liquidation_cascades	0.000140
1	spot_btc_dollar_volume	0.000111
18	futures_eth_total_trades	0.000109
16	futures_eth_coin_volume	0.000084
28	log_returns	0.000078
0	spot_btc_coin_volume	0.000067
4	futures_btc_coin_volume	0.000046
21	futures_eth_liquidations_coin_volume	0.000030
9	futures_btc_liquidations_coin_volume	0.000029
13	spot_eth_total_trades	0.000026
11	spot_eth_coin_volume	0.000024
17	futures_eth_dollar_volume	0.000020
6	futures_btc_total_trades	0.000017
12	spot_eth_dollar_volume	0.000009
23	eth_etf	0.000006
24	btc_etf	0.000003

Goldsmiths University of London¶

MSc Data Science and Artificial Intelligence¶

Final Project in Data Science (2023-24)¶

Supervisor: Dr V L Raju Chinthalapati¶

Author: Alperen Unal - aunal001¶

Student ID: 33804283¶

Phase 1: Data Preprocessing & Feature Engineering¶

1. Utilising Modules¶

2. Loading and Initial Preprocessing The Data¶

2.1. Creating ETF Event Date Data¶

2.2. Cumulative Volume Delta Calculation¶

2.3. Cleaning & Merging ETH and BTC Spot Data¶

2.4. Cleaning & Merging Futures Data¶

2.5. Futures to Spot Price Ratio Data Calculation¶

2.6. Liquidation Cascades¶

2.6.1. Liquidation Cascades Graph¶

3. Feature Selection¶

3.1. Feature Analysis for Log Returns¶

3.1.1. Correlation Coefficient¶

3.1.2. XGBoost Feature Importance¶

3.1.3. Permutation Feature Importance¶

3.1.4. Mutual Information¶

3.2. Feature Analysis for Futures BTC Price¶

3.2.1. Correlation Coefficient¶

3.2.2. XGBoost Feature Importance:¶

3.2.3. Permutation Feature Importance¶

3.2.4. Mutual Information¶

3.3. Detailed Feature Analysis with Visuals for Futures BTC Close Price¶

3.3.1. Futures ETH Close Price¶

futures_eth_close_price is the strongest predictor of futures_btc_close_price, consistently showing high importance across all metrics.¶

3.3.2. Spot BTC CVD¶

Although spot_btc_CVD has a weak direct correlation with futures_btc_close_price, it holds significant importance in the model, particularly in terms of information gain, which suggests it may capture underlying patterns not immediately evident through correlation alone.¶

3.3.3. Futures BTC CVD¶

futures_btc_CVD is a crucial feature with consistent importance across all evaluation metrics, playing a significant role in predicting futures_btc_close_price.¶

3.3.4. Futures BTC Coin Open Interest Close¶

futures_eth_coin_open_interest_close plays a moderate role in predicting futures_btc_close_price, with consistent but not overwhelming importance across evaluation metrics.¶

3.3.5. Spot ETH CVD¶

spot_eth_CVD shows a weak correlation but contributes moderate information, suggesting a hidden importance in the model.¶

3.3.6. Futures ETH Coin Open Interest Close¶

Futures ETH Coin Open Interest Close has an inverse relationship with futures_btc_close_price. However, it plays a significant role in improving model performance.¶

3.3.7. Spot BTC Dollar Volume¶

Spot BTC Dollar Volume has a weak negative relationship with futures_btc_close_price and overall contributes little to the model’s performance.¶

3.3.8. Futures BTC Dollar Volume¶

futures_btc_dollar_volume is a low-importance feature across all metrics, indicating a minimal role in predicting the target variable.¶

3.3.10. Spot ETH Dollar Volume¶

spot_eth_dollar_volume has a weak correlation and provides little value to the model, indicating its low importance across all evaluation metrics.¶

3.3.11. Spot ETH Total Trades¶

spot_eth_total_trades has minimal impact on the model, contributing weakly across all evaluation methods.¶

3.3.12. Futures ETH Total Trades¶

futures_eth_total_trades has negligible importance and little to no influence on the model’s performance.¶

3.3.13. Spot ETH Coin Volume¶

spot_eth_coin_volume holds minimal importance in the model, with a weak correlation and low influence on predictions.¶

14. Spot ETH CVD¶

Futures BTC Coin Volume has a weak negative relationship with futures_btc_close_price and provides a minimal contribution to the model overall.¶

3.3.16. Futures ETH Coin Volume¶

Futures ETH Coin Volume has a weak negative relationship with futures_btc_close_price and offers a minimal contribution to the model overall.¶

3.3.17. Spot BTC Total Trades¶

spot_btc_total_trades has weak predictive power, with moderate importance in XGBoost but minimal impact in other evaluations.¶

3.3.18 Spot BTC Coin Volume¶

While spot_btc_coin_volume has a moderate negative correlation, its overall importance in the model is somewhat limited, although it still contributes meaningful information.¶

3.3.19. Futures ETH CVD¶

futures_eth_CVD is moderately correlated with the target and has high importance in the model, indicating it is a valuable predictor despite not having the strongest direct correlation.¶

3.3.20. Futures BTC CVD¶

Futures BTC CVD has a moderate negative correlation with the BTC futures closing price and plays an important role in the model’s performance, particularly as indicated by XGBoost and Mutual Information.¶

3.3.21. Futures BTC Coin Open Interest Close¶

Futures BTC Coin Open Interest Close has an inverse relationship with futures_btc_close_price and plays a significant role in improving the model’s accuracy.¶

3.3.22. Futures BTC Liquidations Coin Volume¶

Futures BTC Liquidations Coin Volume shows a very weak relationship with the target variable and contributes minimally to the model’s performance, as reflected across all evaluation methods.¶

3.3.23. BTC ETF¶

btc_etf is the least important variable across all metrics, providing minimal predictive value and having virtually no impact on the model’s performance.¶

24. ETH ETF¶

eth_etf is largely irrelevant in predicting futures_btc_close_price, with negligible importance across all evaluation methods.¶

3.3.25. Liquidation Cascades¶

liquidation_cascades provides some predictive value due to its weak positive correlation, but its overall importance is minimal across most metrics.¶

3.3.26. Futures BTC Funding Rate¶

futures_btc_funding_rate has a moderate positive correlation with the target and provides valuable information, though it has a relatively minor impact on model performance.¶

3.3.27. Futures ETH Funding Rate¶

3.3.28. BTC Futures to Spot Price Ratio¶

btc_futures_to_spot shows a moderate positive correlation with the target, suggesting some predictive value, although its importance varies across metrics.¶

3.3.28. ETH Futures to Spot Price Ratio¶